Facilitating data mining on a network of workstations.” in Advances in distributed and parallel knowledge discovery

The computationally-intensive nature of many data mining algorithms and the size of the datasets involved has motivated efforts to use parallel computing to produce timely results. A particularly cost effective computing platform for such paral-lelizations is a network of workstations (NOW). However, there are many problems associated with efficient parallelizations on a NOW, including data transmission over a low bandwidth network, load-balancing, fault-tolerance, interactivity, programming complexity, etc. To address some of these problems, in this paper, we propose the programmable, distributed doall, a generic mechanism, similar to the doall primitive on SMPs, which schedules a set of independent tasks on a NOW. It allows incremental reporting of results, which is used to allow the user to monitor and, if necessary, interrupt the operation. Most importantly, it seeks to reduce communication bandwidth requirements by allowing specification of resource requirements of the tasks at the application programming level. This information is used by a “resource-aware ” scheduler to reduce communication. We evaluate the performance of this mechanism using a template application that resembles, both in its computational and data access patterns, several common data mining applications. We also evaluate the primitive on clustering, a common data mining technique